Method for data processing in a scan microscope comprising a fast scanner and scan microscope comprising a fast scanner

ABSTRACT

The process acquires data blocks in real-time with a fast scanner. The acquired data blocks are then transmitted to a computer system ( 23 ). The data blocks are then processed as a function of a frame burst ratio (N). The transmission of the acquired data blocks to the computer system is a function of the frame burst ratio (N). The frame burst ratio (N) may be either fixed or variable. In any case, optimal utilization of the computer system&#39;s ( 23 ) performance is important. The frame burst ratio (N) is selected by the user or by the computer system ( 23 ) itself as a function of the processing characteristics of the computer system ( 23 ).

The invention concerns a data processing method in a scanning microscopewith a fast scanner.

Furthermore, the invention concerns a scanning microscope with a fastscanner. In particular, the invention concerns a scanning microscopewith a fast scanner, consisting of a scanning module, a position sensor,and at least one detector; with a computer system with at least oneperipheral attached the computer system and one input device. A methodfor setting the image recording of a microscope is disclosed in the notyet published patent application DE 101 34 328.0. Image data istransmitted from an image recording element to a storage element.Control parameters are given to a coder. Coding takes place before thetransmission of image data from the storage element to the computer. Thecoded and transmitted image data are processed by the computer. Theinvention has the disadvantage that a portion of the image data and theinformation contained in it are lost as a result of coding. These dataare permanently lost and unavailable for analysis.

The problem of the invention is to create a method that makes possibledata processing of all data acquired by a fast scanner, without delaysoccurring during processing.

The objective problem is solved by a method having the featurescontained in the characterizing portion of patent claim 1.

A further problem of the invention is to create a scanning microscopewith which all acquired data is processed, without having potentialdelays caused by the computer system, which limit the processing of alldata.

The above problem is solved by a scanning microscope with the featurescontained in the characterizing portion of claim 12.

The invention has the advantage that with the method, data blocks areacquired with a fast scanner. The acquired data blocks are thentransmitted to a computer system. The processing of the data blocks is afunction of the frame burst ratio. In addition, the transmission ofacquired data blocks to the computer system may also be a function of aframe burst ratio. The frame burst ratio must be specified in such a waythat the computer system's performance is optimally utilized. The frameburst ratio is selected by the user, depending on the processingcharacteristics of the computer system and remains constant duringacquisition of the data blocks.

A further advantageous development of the invention is that an adaptivecontrol is envisioned that allows for a variable frame burst ratio. Thecomputer system then adapts the frame burst ratio to the current workingconditions in the computer system, or to the parameter settings of thescanning microscope. When data acquisition begins, the user may enter aninitial value for the frame burst ratio. The computer system may alsoselect an initial value, and adapt it as needed.

The frame burst ratio determines the frequency of the transmitted datablocks or of the on/off ratio, respectively. The computer system adaptsto the performance requirements of the moment by means of the frameburst ratio. All data blocks are first stored in the computer system,and only the data blocks specified by the variable frame burst ratiowill be processed. It is also of advantage that only those data blocksthat correspond to the fixed frame burst ratio specified by the user aretransmitted to and processed by the computer system. The data blocksthat have not yet been transmitted are delayed and then transmitted tothe computer system for processing.

Further advantageous developments of the invention may be inferred fromthe subordinate claims.

The subject of the invention is schematically shown in the diagram andis described on the basis of the following figures. They show:

FIG. 1 a schematic diagram of a scanning microscope with a fast scanner;

FIG. 2 a schematic diagram of the transmission of data from a real-timesystem to a non-real-time system, in which case the frequency of theprocessed data blocks is preselected;

FIG. 3 a block diagram of a first embodiment of the invention, asschematically shown in FIG. 2;

FIG. 4 a schematic diagram of the transmission of data from a real-timesystem to a non-real-time system, in which case the frequency of theprocessed data blocks may be adapted to the performance of the computersystem;

FIG. 5 a block diagram of a second embodiment of the invention, asschematically shown in FIG. 4;

FIG. 6 a schematic diagram of the transmission of data from a real-timesystem to a non-real-time system, in which case the frequency of thesynchronously transmitted data blocks is preselected, and theuntransmitted data at the end of the scan are asynchronously transmittedto the computer system;

FIG. 7 a block diagram of a third embodiment of the invention, asschematically shown in FIG. 6;

FIG. 8 a schematic diagram of the transmission of data from a real-timesystem to a non-real-time system, in which case the frequency of thesynchronously transmitted data blocks is adapted to the computer system,and the untransmitted data at the end of the scan are asynchronouslytransmitted to the computer system; and

FIG. 9 a block diagram of a fourth embodiment of the invention, asschematically shown in FIG. 8.

FIG. 1 is a schematic diagram of an embodiment of the invention in theform of a confocal scanning microscope (100). However, this should notthe viewed as a limitation of the invention. It will be more than clearto the person skilled in the art that the invention may also be realizedwith a conventional microscope or conventional scanning microscope. Theillumination light beam (3) coming from at least one illumination system(1) is reflected by a beam splitter or a suitable reflecting material(5) to a scanning module (7). Before the illumination light beam (3)hits the reflecting material (5) it passes through an illuminationpinhole (6). The scanning module (7) comprises a cardanically suspendedscanning mirror (9), which sends the illumination light beam (3) througha scanner lens (12) and a microscope lens (13) over or through an object(15). When it encounters non-transparent objects (15), the illuminationlight beam (3) is sent over the surface of the object. With biologicalobjects (15) such as samples or transparent objects, the illuminationlight beam (3) can also be sent through the object (15). To this end,nonluminous samples can be prepared with a suitable dye (not shownbecause it is established state of the art). This means that differentfocal planes of the object can be successively scanned by theillumination light beam (3). A position sensor (11) that determines thepositional data of the acquired image data is connected with thescanning module (7). Subsequent combination of the positional data andthe image data yields a 2-dimensional or 3-dimensional frame (or image)of the object (15). The illumination light beam (3) coming from theillumination system (1) is depicted as a solid line. The light comingfrom the object (15) defines a detection light beam (17). This beamreaches the reflecting material (5) through the microscope lens (13) orthe scanner lens (12) via the scanning module (7), passes through it,and by way of a detection pinhole (18) reaches at least one detector(19), which is here embodied as a photo multiplier. The person skilledin the art will understand that other detection components, such asdiodes, diode arrays, CCD chips, or CMOS image sensors can also be used.The detection beam (17) that comes from and is defined by the object(15), respectively, is depicted in FIG. 1 as a broken line. Electricaldetection signals, which are proportional to the light coming from theobject, are generated in the detector (19). The scanning module (7), thepositional sensor (11), and at least one detector (19) together comprisea fast scanner (14). A local storage unit (16), which receives the datafrom at least one detector (19) and the positional sensor (11), isallocated to the fast scanner (14). The data is transferred in suitableform from the local storage unit (16) to a computer system (23). It willbe self-evident to a person skilled in the art-that the position of thescanning mirror (9) can also be determined by the positioning signals.The computer system (23) is designed with at least one peripheral device(27). The peripheral device can, for example, be a display that givesthe user the information needed to adjust the microscope system, or fromwhich he may receive the current setup as well as the image data ingraphic form. Furthermore, the computer system (23) has at least oneinput device such as a keyboard (28), an adjusting device (29) for thecomponents of the microscope system, and a mouse (30). Processing of thedata blocks takes place in at least one peripheral device (27).Processing may be understood as printing, the compression of data,display on a computer screen, on line analysis, or storage in thestorage units.

FIG. 2 presents an embodiment of the method, the so-called frame burstratio, in which only a certain number of data blocks are transmitted.The viewing of a frame all at once (“frame burst”) and not “block byblock” has been known for some time. However, this technique is notsufficient because in fast scanners, the processing of acquired data ina computer system cannot keep up with the acquisition of the dataitself. A local storage unit (16) is allocated to the fast scannerwhich, as previously mentioned, consists of the scanning module (7), theposition sensor (11), and at least one detector (19). The fast scannertransmits the data in real time to its own local storage unit (16) andsimultaneously to the computer system (23) as well. All data blocks(frames 35 ₁, 35 ₂, . . . , 35 _(n)) are stored in the computer system(23). However, only those data blocks that are specified by the frameburst ratio (N) are processed in a peripheral device (27). For example,a frame burst of N=10 means that only every tenth data block isprocessed. The frequency of the data blocks that are processed in thecomputer system (23) is constant. In the example depicted in FIG. 2, theframe burst ratio is N=3 so that every third data block undergoesprocessing in the computer system (23). These data blocks (frames) aredepicted with crosshatches in FIG. 2. It is possible for the user tospecify whatever frame burst ratio (N) (or frequency) he wishes so thatit takes into account the individual imaging characteristics of hiscomputer system (23). The user inputs the frame burst ratio (N) by meansof a keyboard (28), a mouse (30), or an adjusting device (28).

FIG. 3 is a block diagram that depicts the first embodiment of themethod according to the invention. In a first step (40), the data blocks(frames 35 ₁, 35 ₂, . . . , 35 _(n)) acquired by the fast scanner aretransmitted to an internal storage unit in the computer system (23).Subsequently, in a second step (44), the computer system (23) retrievesdata blocks in accordance with the frame burst ratio (N)—for example,every tenth data block—from the storage unit in computer system (23). Afirst decision module (45) tests whether this was the final data block.If the result in this decision module (45) is NO, the process iscontinued in a second decision module (46). Here, the current data blockis tested to determine whether it is a multiple of N. If YES, the datablock is processed or displayed. If the result in the second decisionmodule (46) was NO, the process loops back, the second step (44) isredone, and a corresponding data block is retrieved from the storageunit of the computer system. If the result was YES in the first decisionmodule (45), the rest of the data blocks are retrieved from the storageunit in the computer system and are transmitted for processing and/ordisplay of the other data blocks (49). The user can himself specify theframe burst ratio (N) by means of an input device (47). This then actson the second decision module (46). Depending on the workload of thecomputer system (23) the user can decide whether or not to change theframe burst ratio (N).

FIG. 4 depicts another embodiment of a method in which only a certainnumber of data blocks are processed. As already mentioned in FIG. 2, alocal storage unit (16) is allocated to the fast scanner (14). The fastscanner transmits the data in real time over a dedicated transmissionline (21) to its own local storage unit (16) and simultaneously to thecomputer system (23) as well. All data blocks (frames 35 ₁, 35 ₂, . . ., 35 _(n)) are stored in the computer system (23). However, only thosedata blocks that are specified by the frame burst ratio (N) areprocessed. An adaptive control is envisioned that makes the frame burstratio (N) variable. This means that the frequency of the processed datablocks or of the on/off ratio, respectively, change and adapt to thecurrent performance of the computer system (23). To this end, a feedbacksystem (24) is envisioned between the fast scanner and the computersystem (23) to monitor the delay or acceleration through the computersystem (23) in comparison to the fast scanner, and to readjust thefrequency of the processed data blocks as well.

FIG. 5 is a block diagram that depicts the second embodiment of themethod according to the invention. According to the method depicted inFIG. 3, the user of the confocal scanning microscope (100) mustdetermine the setting of the frame-burst ratio experimentally and set itaccordingly. The processing characteristics of the confocal scanningmicroscope (100) are determined by a variety of peripheral conditionsand is not deterministic. They may change over time and lead todeceleration or acceleration of processing as the number of framesincreases. For this reason it is essential to be able to readjust theframe burst ratio adaptively for optimal display or processing of thescan data. Here, processing by the computer system (23) is continuallymonitored during data acquisition. Any potential delay or accelerationis then countered by an increase or decrease in the frame burst ratio.Similar to the process depicted in FIG. 3, the data blocks (frames 35 ₁,35 ₂, . . . , 35 _(n)) acquired by the fast scanner are transmitted toan internal storage unit in the computer system (23) in a first step(40). Subsequently, the computer system (23) retrieves the frames ordata blocks, respectively, from the storage unit in the computer system(23) in a second step (44) in accordance with an initial frame burstratio (N), such as every tenth data block. A first decision module (45)tests whether this was the final data block. If the result in thisdecision module (45) is NO, the process is continued in a seconddecision module (46). Here, the current data block is tested todetermine whether it is a multiple of N. If YES, the data block isprocessed (48). If the result in the second decision module (46) was NO,the process loops back, the second step (44) is redone, and acorresponding data block is retrieved from the storage unit in thecomputer system. If the result was YES in the first decision module(45), the rest of the data blocks are retrieved from the storage unit ofthe computer system and are transmitted for processing of the other datablocks (49). A feedback module (50) connects the first step (40) withthe second decision module (46). As a result, the frame burst ratio (N)can be adapted to the momentary performance of the computer system (23).

FIG. 6 is a schematic representation of a method for the partiallysynchronous transmission, analysis, and display of a scanner's acquireddata. The difference between it and the depiction in FIG. 2 is that inthe computer system (23) only the frames chosen by the user inaccordance with the prespecified frame burst ratio (N) are synchronouslytransmitted and processed. The fast scanner transmits the data in realtime to its own local storage unit (16), and at the same time only theframes or data blocks, respectively that correspond to the fixed frameburst ratio specified by the user are transmitted to the computer system(23). The transmitted data blocks (frames 35 _(N-(N-1)), 35 _(2N-(N-1)),. . . , 35 _(NN-(N-1))) are processed immediately in the computer system(23). The frequency of the data blocks that are processed in thecomputer system (23) is constant. The data blocks (frames) that areprocessed in the computer system (23) are depicted with crosshatches inFIG. 6. It is possible for the user to specify whatever frame burstratio (N) (or frequency) he wishes so that it takes into account theindividual processing characteristics of his computer system (23). Theuser must specify the processing characteristics beforehand. The userinputs the frame burst ratio (N) as described in FIG. 2. Theasynchronously transmitted data blocks remain in the scanner's storage(16) and are only transmitted to the computer system (23) with a delayafter the end of scanning from the scanner and are therefore processedwith a delay. The asynchronously transmitted data blocks are identifiedin FIG. 6 as empty boxes. The delayed transmission is represented by thedashed arrows on the schematically depicted data blocks. In this way,data recording by the PC that is too slow, caused by too narrowtransmission bandwidth, lack of storage space, or system overloadresulting from other running processes, can be avoided.

A block diagram of the third embodiment of the invention as depictedschematically in FIG. 6 is shown in FIG. 7. FIG. 7 shows a block diagramthat describes the third embodiment of the method according to theinvention. In a first step (40) the data blocks (frames 35 _(N-(N-1)),35 _(2N-(N-1)), . . . , 35 _(NN-(N-1))) that were specified by the useraccording to the frame burst ratio (N) are transmitted to the internalstorage unit of the computer system (23) and immediately processed inthe computer system (23). A first decision module (45) tests whetherthis was the final data block in accordance with the constant frameburst ratio. If the result in this decision module (45) is NO, theprocess is continued in a second decision module (46). Here, the currentdata block is tested to determine whether it is a multiple of N. If YES,the data block is transmitted (52) to the computer system (23) andprocessed (54) in the computer system (23). If the result in the seconddecision module (46) was NO, the process loops back, the second step(44) is redone, and a corresponding data block is retrieved from thestorage unit and transmitted to the computer system (23). If the resultwas YES in the first decision module (45), the rest of the data blocksthat have not yet been transmitted to the computer system are sentdirectly from the fast scanner to the computer system (23). Delayedprocessing (56) of the rest of the data blocks that do not correspond tothe frame burst ratio (N) then occurs in the computer system (23).

FIG. 8 shows a schematic representation of a method that is very similarto the method in FIG. 6. The method shown in FIG. 8 differs to theextent that here there is no fixed frame burst ratio (N), but rather aframe burst ratio that adapts to the current performance characteristicsof the computer system (23). First of all, a frame burst ratio (N) canbe specified by the user. The fast scanner transmits the data in realtime to its own local storage unit (16), while at the same time only theframes or data blocks that correspond to the fixed frame burst ratio (N)specified by the user are transmitted to the computer system (23). Thetransmitted data blocks (frames 35 _(N-(N-1)), 35 _(2N-(N-1)), . . . ,35 _(NN-(N-1))) are processed immediately in the computer system (23).An adaptive control is envisioned that makes the frame burst ratio (N)variable. This means that the frequency of the processed data blocks orof the on/off ratio, respectively, change and adapt to the currentperformance of the computer system (23). To this end, a feedback system(24) to monitor the delay or acceleration by the computer system (23) incomparison to the fast scanner is envisioned between the fast scannerand the computer system (23) and readjusts it as needed. This changesthe frequency of the synchronously transmitted and processed data blocks(frames 35 _(N-(N-1)), 35 _(2N-(N-1)), . . . , 35 _(NN-(N-1))).

A block diagram of the fourth embodiment of the invention asschematically depicted in FIG. 8 is shown in FIG. 9. FIG. 9 shows ablock diagram that describes the fourth embodiment of the methodaccording to the invention. In a first step (40), the data blocks(frames 35 _(N-(N-1)), 35 _(2N-(N-1)), . . . , 35 _(NN-(N-1))) that werespecified by the user in accordance with the frame burst ratio (N) aretransmitted to the internal storage unit in the computer system (23) andimmediately processed in the computer system (23). A first decisionmodule (45) tests whether this was the final data block in accordancewith the adaptable frame burst ratio. If the result in this decisionmodule (45) is NO, the process is continued in a second decision module(46). Here, the current data block is tested to determine whether it isa multiple of N. If YES, the data block is transmitted (52) to thecomputer system (23) and then stored and processed (54) in the computersystem (23). A feedback module (60) is connected to the computer system(23) in which the storage and processing (54) of the data block takesplace.

If the result in the second decision module (46) was NO, the processloops back, the second step (44) is redone, and a corresponding datablock is retrieved from the storage unit and transmitted to the computersystem (23). If the result was YES in the first decision module (45),the rest of the data blocks that have not yet been transmitted to thecomputer system are sent directly to the computer system (23).Asynchronous processing (56) of the rest of the data blocks that do notcorrespond to the frame burst ratio (N) then occurs in the computersystem (23). In this way, the frame burst ratio (N) can be adapted tothe current performance of the computer system (23). As a result, datarecording by the computer system (23) that is too slow, caused by a toonarrow transmission bandwidth, lack of storage space, or system overloadresulting from other running processes, can be avoided.

In both the second and the fourth embodiment of the invention, the frameburst ratio (N) is adapted to the conditions of the computer system(23). In addition to the provision for the user to specify an initialvalue, the computer system (23) itself can specify an initial value forthe frame burst ratio (N). It is selected automatically during the dataacquisition cycle by means of a suitable algorithm, depending on the setscan speed and whatever processing routines are running in the computersystem (23). The computer system (23) constantly monitors the differencebetween the data blocks acquired by the scanner and the number of datablocks actually processed by the computer system (23). The differencebetween the acquired and the processed data blocks is determined, andappropriate action is initiated. Optimally, the difference should be 0in a computer system running at full load. Adaptive control aims atmaintaining his condition.

If the difference increases, the computer system (23) is not capable ofmeeting the required frame burst ratio (N). The frame burst ratio (N) isincreased. If the difference is not zero (0) and the computer system(23) is not operating at full load, the computer system (23) is able tomeet the required frame burst ratio (N) beyond the extent required. Inthis case, frame burst ratio (N) will decrease.

All obtained values for the frame burst ratio (N) of the dataacquisition cycle are summarized using a suitable algorithm into anempirical value in the computer system (e.g., a mean or average value,in the simplest case). This empirical value is stored in the computersystem so that it is available to specify a more optimal initial valuefor later comparable data acquisition cycles.

The following example is meant to illustrate the adaptive control of theframe burst ratio (N) in a data acquisition cycle. To start with, theframe burst ratio (N) is preset to 10. The difference increases to 50,and the frame burst ratio (N) is increased to 20. During the subsequentdata acquisition cycle, the difference falls to 0, and when the computersystem (23) is running at full load, the frame burst ratio (N) is set at20. The user changes the scan parameters during data acquisition. Thedifference increases to 100. As a consequence, the frame burst ratio (N)is increased to 30. The scan parameters are changed, and the differencefalls below zero when the computer system (23) is not running at fullload. As a consequence, the frame burst ratio (N) is decreased to 20.The computer system (23) is disturbed by an external process. As aresult, the difference again increases to 50. The frame burst ratio (N)is then increased to 25. These changes in the frame burst ratio (N)continue during the entire data acquisition cycle.

1. Data processing method in a scanning microscope with a fast scanner,characterized by the following steps: acquisition of data blocks inreal-time with a fast scanner; transmission of the acquired data blocksto a computer system (23); and processing of the data blocks as afunction of a frame burst ratio (N).
 2. Method according to claim 1,characterized in that the transmission of the acquired data blocks is afunction of the frame burst ratio (N), in which case the frame burstratio (N) is selected such that optimal utilization of the computersystem's (23) performance ensues.
 3. Method according to claim 2,characterized in that the frame burst ratio (N) is selected by the useras a function of the processing characteristics of the computer system,and in that it remains constant during acquisition of the data blocks.4. Method according to claim 3, characterized in that all data blocks(35 ₁, 35 ₂, . . . , 35 _(n)) are stored in the computer system (23),and in that those data blocks that are specified by the constant frameburst ratio (N) are processed.
 5. Method according to claim 1,characterized in that adaptive control is envisioned that makes theframe burst ratio (N) variable.
 6. Method according to claim 5,characterized in that an initial value is specified for the frame burstratio (N) at the start of data acquisition.
 7. Method according to claim6, characterized in that the frame burst ratio (N) determines thefrequency of the transmitted data blocks and of the on/off ratio,respectively, and is adapted to the current performance of the computersystem (23); in that all data blocks (35 ₁, 35 ₂, . . . , 35 _(n)) arestored in the computer system (23); and in that those data blocks thatare specified by the variable frame burst ratio (N) are processed. 8.Method according to claim 1, characterized in that the frame burst ratio(N) is selected by the user as a function of the processingcharacteristics of the computer system (23) and remains constant duringacquisition of the data blocks, and in that at the same time only thosedata blocks that correspond to the fixed frame burst ratio (N) specifiedby the user are transmitted to the computer system (23) and areprocessed by the computer system (23).
 9. Method according to claim 8,characterized in that the data blocks that have not yet been transmittedare transmitted to the computer system (23) with a delay and are thenprocessed.
 10. Method according to claim 1, characterized in that theframe burst ratio (N) is selected as a function of the processingcharacteristics of the computer system (23) and are adapted by thecomputer system during acquisition of the data blocks; and in that atthe same time only those data blocks that correspond to the variableframe burst ratio (N) are transmitted to the computer system (23). 11.Method according to claim 10, characterized in that the data blocks thatdo not correspond to the variable frame burst ratio (N) are transmittedand/or processed to the computer system (23) with a delay.
 12. Scanningmicroscope with a fast scanner, consisting of a scanning module (7), aposition sensor (11), and at least one detector (19), with a computersystem (23) with at least one peripheral device (27) allocated to thecomputer system (23), and with one input device (25), characterized inthat a local storage unit (16) is allocated to the fast scanner; in thatdata blocks are transmissible from the local storage units (16) of thefast scanner to the computer system (23), in which case a frame burstratio (N) is selected such that optimal utilization of the computersystem's (23) performance is achieved, and in that the transmitted datablocks that are a function of the frame burst ratio (N) can be processedin a peripheral device (27).
 13. Scanning microscope according to claim12, characterized in that data blocks are transmissible to the computersystem (23) as a function of the frame burst ratio (N).
 14. Scanningmicroscope according to claim 12, characterized in that the frame burstratio (N) is constant during a data acquisition cycle and reflects theprocessing characteristics of the computer system (23).
 15. Scanningmicroscope according to claim 12, characterized in that adaptive controlof the computer system (23) is envisioned that adapts the frame burstratio (N) during the data acquisition cycle.
 16. Scanning microscopeaccording to claim 12, characterized in that the frame burst ratio (N)is constant during a data acquisition cycle and reflects the processingcharacteristics of the computer system (23), and in that the computersystem (23) receives first the data blocks that correspond to a fixedframe burst ratio (N).
 17. Scanning microscope according to claim 12,characterized in that adaptive control is envisioned that adapts theframe burst ratio (N) during a data acquisition cycle, and in that thecomputer system (23) receives first the data blocks that correspond tothe variable frame burst ratio (N).